Over the last decade, cybersecurity attacks have become a pervasive problem for internet users as many network devices and other resources have been subject to attack and compromised. In particular, a technique known as “heap spray” allows malicious software sometimes referred to as malware or an exploit residing within a network device to perpetrate a cybersecurity attack. During a heap spray attack, an exploit inserts a portion of code, for example, a sequence of No Operation (NOP) instructions and a sequence of bytes at a random location within process address space of the network device. The sequence of bytes may be directed at performing malicious activity, wherein the sequence of bytes may be in the form of code, commonly referred to as “shellcode.” This location may a memory location within a heap (e.g., a predetermined amount of virtual memory allocated for the software). Once shellcode has been inserted into the heap, the exploit may be triggered during processing which causes the application to execute the shellcode thereby allowing the exploit to assume control of the execution of code within the memory of the network device. This process of assuming control of execution enables the attacker to execute instructions on the network device which may be malicious or anomalous to the network device, to its content or to the owner of the network device. Heap spray attacks use chunks of No Operation (NOP) instructions, also known as NOP sleds, to facilitate the execution of the shellcode. By orienting a NOP sled at an address just prior to shellcode, the execution flow of the processing unit of the network device is quickly directed to the shellcode.
The NOP instructions may be a sequence of known instructions (e.g., “patterns”). Conventional heap spray detection methods attempt to identify heap spray attacks by detecting the NOP sleds and comparing the sequence of NOP instructions to NOP instructions appearing on a list of permitted (and non-malicious) NOP instructions. Such a list may be referred to as a “whitelist” of NOP instructions. Those sequences of NOP instructions found on the whitelist are dismissed as non-malicious. However, those sequences of NOP instructions found in the heap but not found on the whitelist are flagged as a probable NOP sled portion of a heap spray attack. Alternatively, sequences of NOP instructions may be compared to a “blacklist” of NOP instructions, wherein NOP instructions found on the blacklist may be considered malicious.
Several issues with a detection strategy using a whitelist and/or blacklist of NOP instructions exist. First, comparing sequences of NOP instructions to NOP instructions appearing on a whitelist may result in a high number of false positives as a whitelist rarely contains all possible permitted or non-malicious NOP instructions. Several non-malicious NOP instructions not appearing on the whitelist may reside in received network traffic. This in turn prompts the detection system to improperly flag one or more objects within the network traffic as containing or, viewed differently, instigating, when processed, a heap spray attack and return a false positive to a network administrator. Second, as false positives are reported, the NOP instruction that caused the false positive is typically added to the whitelist in order to prevent future false positives based on that particular non-malicious NOP instruction. However, a whitelist of all possible non-malicious NOP instructions is untenable because such as list would require constant updating.
Similarly, numerous false positives may result from comparisons with a blacklist. For instance, a sequence of instructions may differ in an inconsequential manner (e.g., changing of one or two instructions) from a NOP sequence appearing on a blacklist, wherein the difference results in the sequence of instructions not being flagged as malicious while performing the same function when executed as a NOP sequence appearing on the blacklist.
Therefore, the efficacy of heap spray detection systems using comparisons to either blacklists or whitelists is limited by the number of known malicious and non-malicious patterns, respectively. As a large number of permutations exists wherein one or more instructions or bytes are altered (e.g., order of one or more instructions swamped, or one or more instructions replaced with one or more alternative instructions) such that the alteration to the pattern does not change the functionality of the pattern, such heap spray detection systems may prove to be ineffective.
However, if too many patterns are used in a heap spray detection algorithm, the detection process may take a considerable amount of time and fail to return a result within the allotted time for the whitelist or blacklist comparison. Furthermore, while the detection process is attempting to perform such a comparison, a heap spray attack may be executing and consuming a large amount of memory and CPU resources and; therefore, further impede the speed of the detection process.
Another disadvantage to current heap spray detection systems is that numerous patterns appearing on blacklists as NOP sequences are common in non-malicious objects, resulting in false positives. For example, non-malicious PowerPoint objects may include several large regions of the same instructions that may trigger a comparison with a pattern on a blacklist. However, the large regions of the same instructions may merely represent the background of the slide. Therefore, current detection systems may flag this non-malicious PowerPoint object resulting in a false positive.
Additionally, content appearing as part of a heap spray attack is not always consistent within a memory heap, thus, the location of such content is rarely predictable. Therefore, heap spray detection systems that rely on scanning at predetermined locations for predetermined patterns may encounter numerous false negatives.